Release Notes - 04 April 2023 Updates
Explore the new features added in this update!
Updated on: 04 April 2023
Release Version: 0.12
| Feature | Description |
|---|---|
| Code for custom transformation jobs is now generated by Lazsa Data Pipeline Studio based on the SQL queries provided by the user. | Lazsa Data Pipeline Studio provides templatized and custom transformation jobs for Snowflake. For custom transformation jobs, the User Interface of Lazsa Data Pipeline Studio provides the option to create SQL queries by selecting specific columns of tables. Lazsa combines the SQL queries along with the transformation logic to generate the code for custom transformation jobs. |
| Filtering of columns is now supported in Databricks transformation and integration and creating custom columns is supported in Databricks integration. | You can now filter columns of tables from the data source during data integration in Lazsa Data Pipeline Studio. The filtered columns are then pushed to the target data lake. You can also create custom column names and map them to the default columns in source before pushing them to a table in the target. The support is currently available for CSV format. |
| Filtering and editing join queries is now supported in a Databricks transformation job. | You can now add additional filters in the join queries of a Databricks transformation job. Lazsa Data Pipeline Studio also lets you edit a join query and create a base version of the query. You can update a base version by editing a query and saving the job configuration. |
| Data Validator now supports IsContainedIn constraint for integer, double, and timestamp data types. | The Data Validator node of Data Quality stage now supports the IsContainedIn constraint for data types like integer, double, and timestamp. Earlier the support was available for string data type. |
| Multi-column selection of constraints now enabled in Data Analyzer and Data Validator. |
You can create a composite key in Data Analyzer and Data Validator of the Data Quality stage in Lazsa Data Pipeline Studio. The following constraints are supported for multi-column selection: Data Analyzer:
Data Validator:
|
| File-level filtering is now supported for Amazon S3 source files. | Lazsa Data Pipeline Studio now supports filtering of Amazon S3 source files by using regex to include or exclude files that are sent to the data integration stage. |
| Databricks Transformation now supports adding custom columns with system or static parameters for join, union, and aggregate functions. | You can now add custom columns with static or system parameters to join, union, and aggregate functions in a Databricks Transformation job in Lazsa Data Pipeline Studio. |